Laplace's Rule of Succession in Information Geometry

When observing data x1, . . . , xt modelled by a probabilistic distribution pθ(x), the maximum likelihood (ML) estimator θML = argmaxθ ∑︀t i=1 ln pθ(xi) cannot, in general, safely be used to predict xt+1. For instance, for a Bernoulli process, if only “tails” have been observed so far, the probability of “heads” is estimated to 0. Laplace’s famous “add-one” rule of succession (e.g., [Grü07]) regularizes θ by adding 1 to the count of “heads” and of “tails” in the observed sequence. Bayesian estimators suffer less from this problem, as every value of θ contributes, to some extent, to the Bayesian prediction of xt+1 knowing x1:t. However, their use can be limited by the need to integrate over parameter space or to use Monte Carlo samples from the posterior distribution. For Bernoulli distributions, Laplace’s rule is equivalent to using a uniform prior on the Bernoulli parameter. The non-informative Jeffreys prior on the Bernoulli parameter corresponds to Krichevsky and Trofimov’s “add-one-half” rule [KT81]. Thus, in this case, some Bayesian predictors have a simple implementation. We claim (Theorem 1) that for exponential families1, Bayesian predictors can be approximated by mixing the ML estimator with the sequential normalized maximum likelihood (SNML) estimator from universal coding theory [RSKM08, RR08], which is a fully canonical version of Laplace’s rule. The weights of this mixture depend on the density of the desired Bayesian prior with respect to the non-informative Jeffreys prior, and are equal to 1/2 for the Jeffreys prior, thus extending Krichevsky and Trofimov’s result. The resulting mixture also approximates the “flattened” ML estimator from [KGDR10]. Thus, it is possible to approximate Bayesian predictors without the cost of integrating over θ or sampling from the posterior. The statements below emphasize the special role of the Jeffreys prior and the Fisher information metric. Moreover, the analysis reveals that the direction of the shift from the ML predictor to Bayesian predictors is systematic and given by an intrinsic, information-geometric vector field on statistical manifolds. This could contribute to regularization procedures in statistical learning.